Image processing apparatus, image processing method, and image processing program

ABSTRACT

An image processing apparatus includes an imaging unit that images a user and a real object, an analysis unit that analyzes an attitude of the real object based on imaging information captured by the imaging unit, and a control unit that controls display of an image related to the real object based on the attitude of the real object.

FIELD

The present invention relates to an image processing apparatus, an imageprocessing method, and an image processing program.

BACKGROUND

Various techniques for displaying a three-dimensional image on a displayhave been proposed. Therefrom, proposals have also been made forautostereoscopic display without using a tool such as glasses. Forexample, as a display related to the autostereoscopic display, there isa Light Field display represented by a lenticular method.

In displaying a three-dimensional image on the Light Field display,right and left viewpoint positions of a user are detected, optimal lightrays are collected at the viewpoint positions, and an image for theright eye and an image for the left eye are generated.

CITATION LIST Patent Literature

-   Patent Literature 1: JP 2000-224612 A-   Patent Literature 2: JP 2018-44824 A-   Patent Literature 3: JP H9-54821 A

SUMMARY Technical Problem

However, in the above conventional technique, there is room forimproving a process of displaying the three-dimensional image.

Therefore, the present disclosure provides an image processingapparatus, an image processing method, and an image processing programthat are configured to appropriately display a three-dimensional imagefor the user.

Solution to Problem

To solve the problems described above, an image processing apparatusaccording to an embodiment of the present disclosure includes: animaging unit that images a user and a real object; an analysis unit thatanalyzes an attitude of the real object based on imaging informationcaptured by the imaging unit; and a control unit that controls displayof an image related to the real object based on the attitude of the realobject.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an example of the appearance of animage processing apparatus according to a first embodiment.

FIG. 2 is a diagram illustrating an example of three-dimensional imagedisplay by the image processing apparatus according to the firstembodiment.

FIG. 3 is a diagram illustrating an example of a situation according tothe first embodiment.

FIG. 4 is a diagram illustrating a configuration example of the imageprocessing apparatus according to the first embodiment.

FIG. 5 is a diagram illustrating the processing by an analysis unitaccording to the first embodiment.

FIG. 6 is a table illustrating an example of a data structure of ananalysis result table.

FIG. 7 is a flowchart illustrating a process procedure performed by theimage processing apparatus based on a control policy P1.

FIG. 8 is a diagram illustrating processing performed by a control unitbased on a control policy P2.

FIG. 9 is a flowchart illustrating a process procedure performed by theimage processing apparatus based on the control policy P2.

FIG. 10 is a diagram illustrating processing performed by the controlunit based on a control policy P3.

FIG. 11 is a flowchart illustrating a process procedure performed by theimage processing apparatus based on the control policy P3.

FIG. 12 is a diagram illustrating processing performed by the controlunit based on the control policy P3.

FIG. 13 is a flowchart illustrating a process procedure performed by theimage processing apparatus based on a control policy P4.

FIG. 14 is a diagram (1) illustrating processing performed by thecontrol unit based on a control policy P5.

FIG. 15 is a diagram (2) illustrating processing performed by thecontrol unit based on the control policy P5.

FIG. 16 is a flowchart illustrating a process procedure performed by theimage processing apparatus based on the control policy P5.

FIG. 17 is a diagram (1) illustrating processing performed by thecontrol unit based on a control policy P6.

FIG. 18 is a diagram (2) illustrating processing performed by thecontrol unit based on the control policy P6.

FIG. 19 is a flowchart illustrating a process procedure performed by theimage processing apparatus based on the control policy P6.

FIG. 20 is a diagram illustrating processing performed by the controlunit based on a control policy P7.

FIG. 21 is a flowchart illustrating a process procedure performed by theimage processing apparatus based on the control policy P7.

FIG. 22 is a diagram illustrating processing performed by the controlunit based on a control policy P8.

FIG. 23 is a flowchart illustrating a process procedure performed by theimage processing apparatus based on the control policy P8.

FIG. 24 is a diagram illustrating an example of a situation according toa second embodiment.

FIG. 25 is a diagram (1) illustrating a configuration example of animage processing apparatus according to the second embodiment.

FIG. 26 is a diagram (2) illustrating a configuration example of animage processing apparatus according to the second embodiment.

FIG. 27 is a flowchart (1) illustrating a process procedure performed bythe image processing apparatus according to the second embodiment.

FIG. 28 is a flowchart (2) illustrating a process procedure performed bythe image processing apparatus according to the second embodiment.

FIG. 29 is a diagram illustrating an example of a situation according toa third embodiment.

FIG. 30 is a diagram illustrating a configuration example of an imageprocessing apparatus according to the third embodiment.

FIG. 31 is a flowchart illustrating a process procedure performed by aterminal device according to the third embodiment.

FIG. 32 is a flowchart illustrating a process procedure performed by theimage processing apparatus according to the third embodiment.

FIG. 33 is a diagram illustrating an example of a shielded area and anon-shielded area when there is a plurality of users.

FIG. 34 is a flowchart illustrating a process procedure performed by animage processing apparatus according to other embodiments.

FIG. 35 is a hardware configuration diagram illustrating an example of acomputer implementing the functions of the image processing apparatus.

DESCRIPTION OF EMBODIMENTS

The embodiments of the present disclosure will be described in detailbelow with reference to the drawings. Note that in the followingembodiments, the same portions are denoted by the same referencenumerals or symbols, and a repetitive description thereof will beomitted.

Furthermore, the present disclosure will be described in the order ofthe items shown below.

1. First Embodiment 1.1. Example of Appearance of Image ProcessingApparatus 1.2. Situation According to First Embodiment 1.3.Configuration of Image Processing Apparatus According to FirstEmbodiment 1.4. Effects of Image Processing Apparatus According to FirstEmbodiment 2. Second Embodiment 2.1. Situation According to SecondEmbodiment 2.2. Configuration of Image Processing Apparatus According toSecond Embodiment 2.3. Effects of Image Processing Apparatus Accordingto Second Embodiment 3. Third Embodiment 3.1. Situation According toThird Embodiment 3.2. Configuration of Image Processing ApparatusAccording to Third Embodiment] 3.3. Effects of Image ProcessingApparatus According to Third Embodiment 4. Other Embodiments

5. Hardware configuration

6. Conclusion

1. First Embodiment «1.1. Example of Appearance of Image ProcessingApparatus»

FIG. 1 is a diagram illustrating an example of the appearance of animage processing apparatus according to a first embodiment. An imageprocessing apparatus 100 includes a base 2 and a display 3 that israised upward from the base 2. The image processing apparatus 100includes a camera (stereo camera) 4 on the upper side of the display 3.The camera 4 is configured to images a “user” in front of the display 3and a “real object” held by the user.

The image processing apparatus 100 is configured to display athree-dimensional image using, for example, a lenticular method on thedisplay 3. Roughly speaking, viewpoint positions of the user's nakedeyes wearing no glasses or the like for stereoscopic display is detectedusing images captured by the camera 4. Images (parallax images) for theright and left eyes are generated with light rays collected at therespective viewpoint positions on the right and left sides, and thegenerated images are displayed on the display 3 on which a lenticularlens is mounted.

FIG. 2 is a diagram illustrating an example of three-dimensional imagedisplay by the image processing apparatus according to the firstembodiment. As illustrated in FIG. 2 , the image processing apparatus100 detects the viewpoint positions (left eye 5 a, right eye 5 b) of theuser on the basis of the images captured by the camera 4, and generatesimages (parallax images) for the right and left eyes with light rayscollected at the respective viewpoint positions on the right and leftsides. The image processing apparatus 100 displays the generated imageson the display 3, and thus, the user can view a three-dimensional image6 without using glasses, a head up display (HUD), or the like.

«1.2. Situation According to First Embodiment»

FIG. 3 is a diagram illustrating an example of a situation according tothe first embodiment. As illustrated in FIG. 3 , it is assumed that auser 10 is in front of the image processing apparatus 100, and the user10 holds a real object 11 a. The image processing apparatus 100 capturesan image including the user 10 and the real object 11 a by the camera 4.The image processing apparatus 100 generates an image (virtual object 11b) related to the real object 11 a on the basis of the image (imaginginformation) captured by the camera 4, causing the display 3 to displaythe three-dimensional image of the virtual object 11 b thereon. When theuser 10 moves the real object 11 a, the image processing apparatus 100controls the movement of the virtual object 11 b, according to themovement of the real object 11 a.

Note that the real object 11 a and the virtual object 11 b may not becompletely the same in shape. For example, the real object 11 a may be apart (only a face or the like) of the virtual object 11 b, and the shapeof the real object 11 a may be an approximate shape of the virtualobject 11 b. Note that the real object 11 a may be a non-rigid object,and the user 10 may deform part of the shape of the real object 11 a.

«1.3. Configuration of Image Processing Apparatus According to FirstEmbodiment»

FIG. 4 is a diagram illustrating a configuration example of the imageprocessing apparatus according to the first embodiment. As illustratedin FIG. 4 , the image processing apparatus 100 is connected to an imagecapturing unit 50 and an image display unit 60. Note that the imageprocessing apparatus 100 may include the image capturing unit 50 and theimage display unit 60.

The image capturing unit 50 is a stereo camera imaging the user 10 andthe real object 11 a held by the user 10 that are within an imagingrange. The image capturing unit 50 corresponds to the camera 4illustrated in FIGS. 1 to 3 . The image capturing unit 50 includes, forexample, another image sensor such as a complementary metal oxidesemiconductor (CMOS) sensor or a charge coupled device (CCD). The imagecapturing unit 50 outputs information about an image captured to theimage processing apparatus 100. In the following description, theinformation about the image captured by the image capturing unit 50 isreferred to as the “imaging information”.

Since the image capturing unit 50 is the stereo camera, the imaginginformation includes an image captured by one of the cameras and animage captured by the other camera. The image capturing unit 50 capturesan image at a predetermined frame rate (frame per second (fps)), andoutputs the imaging information to the image processing apparatus 100every time an image is captured. Frame numbers may be assigned to theimaging information in chronological order in ascending order.

The image display unit 60 is a display device that displays athree-dimensional image on the basis of information output from theimage processing apparatus 100. The image display unit 60 corresponds tothe display 3 illustrated in FIGS. 1 to 3 . For example, the imagedisplay unit 60 displays the three-dimensional image of the virtualobject 11 b illustrated in FIG. 3 .

The image processing apparatus 100 includes a viewpoint positiondetection unit 110, a display image generation unit 120, an analysisunit 130, a storage unit 140, and a control unit 150.

The viewpoint position detection unit 110 is a processing unit thatperforms face detection on the basis of the imaging information anddetects the viewpoint positions of the user 10. The viewpoint positionsdetected by the viewpoint position detection unit 110 include thepositions of the left and right eyes of the user 10. The viewpointposition detection unit 110 outputs viewpoint position information tothe display image generation unit 120. For a method of detecting theuser's face (face frame, coordinates of the left eye, and coordinates ofthe right eye) by the viewpoint position detection unit 110, a knownmethod such as a method using image features is applicable. For example,the coordinates of the left eye and the coordinates of the right eye aretwo-dimensional coordinates.

The display image generation unit 120 is a processing unit thatgenerates information for causing the image display unit 60 to displaythe three-dimensional image and outputs the generated information to theimage display unit 60. For example, the display image generation unit120 acquires the viewpoint position information from the viewpointposition detection unit 110, and transforms the two-dimensionalcoordinates indicating the viewpoint positions to viewpoint coordinates(three-dimensional coordinates) at a spatial position by applying aknown method.

The display image generation unit 120 generates light ray (image)information on the basis of a relationship between information about thethree-dimensional image determined by both model data 140 a and controlinformation and the viewpoint coordinates at the spatial position,outputs the light ray (image) information to the image display unit 60to cause the image display unit 60 to display the light ray (image),thereby showing the three-dimensional image to the user 10.

Here, the model data 140 a is information in which the shape, color,reference size, and the like of each portion of the virtual object 11 bare defined in advance, and is stored in the storage unit 140. Thedisplay image generation unit 120 acquires the model data 140 a from thestorage unit 140.

The control information includes an attitude and display coordinates.The attitude is the attitude of the virtual object 11 b. The attitudemay be defined in any manner. For example, a difference from a referenceposition and a reference direction of each portion is specified as theattitude. The display coordinates indicate coordinates for displayingthe virtual object 11 b on a display screen of the image display unit60. The display image generation unit 120 acquires the controlinformation from the control unit 150 which is described later.

The analysis unit 130 is a processing unit that performs variousanalyses on the basis of the imaging information acquired from the imagecapturing unit 50 and outputs results of the analyses to the controlunit 150. Hereinafter, various analysis processes performed by theanalysis unit 130 will be described.

FIG. 5 is a diagram illustrating the processing by the analysis unitaccording to the first embodiment. In FIG. 5 , the camera 4 correspondsto the image capturing unit 50, and the display 3 corresponds to theimage display unit 60. It is assumed that the camera 4 is installed onthe upper side of the display 3, and a distance between the display 3and the user 10 and a distance between the camera 4 and the user 10 arethe same. It is assumed that internal parameters of the camera 4 and aninstallation position (external parameter) of the camera 4 are known.

An example of “processing of detecting the real object 11 a” by theanalysis unit 130 will be described. The analysis unit 130 performsstereo matching for each portion (pixel) of the two images included inthe imaging information to calculate a distance between the respectivepixels. The analysis unit 130 detects, as the real object 11 a, a regionof pixels having a distance from the display 3 that is less than athreshold and having a difference in distance that is less than athreshold. The analysis unit 130 may detect the real object 11 a fromthe imaging information by using any other known technique.

An example of “a process of calculating the attitude of the real object11 a” by the analysis unit 130 will be described. The analysis unit 130detects a region of the real object 11 a from the imaging information bythe processing of detecting the real object 11 a, and identifies theposition of each portion of the real object 11 a. The analysis unit 130calculates, as the attitude of the real object 11 a, a differencebetween the identified position of each portion and the referenceposition and reference direction of each portion set in advance.

An example of “a process of calculating the size of the real object 11a” by the analysis unit 130 will be described. The analysis unit 130detects the region of the real object 11 a from the imaging informationthrough the processing of detecting the real object 11 a The analysisunit 130 uses a conversion table in which a relationship between a sizeof a region in the imaging information and a size in a real space isdefined, and calculates the size of the real object 11 a in the realspace from the size of the region of the real object 11 a. For example,the analysis unit 130 sets a vertical width H and a horizontal width Wof the real object 11 a in the real space, as the size of the realobject 11 a.

An example of “a process of calculating the position of the real object11 a” by the analysis unit 130 will be described. The analysis unit 130detects the region of the real object 11 a from the imaging informationthrough the processing of detecting the real object 11 a The analysisunit 130 detects the center of gravity of the region of the real object11 a as the coordinates of the real object 11 a. The analysis unit 130uses a conversion table in which a relationship between coordinates(two-dimensional coordinates) in the imaging information and a position(three-dimensional coordinates) in the real space is defined, andcalculates the position of the real object 11 a in the real space fromthe coordinates in the imaging information.

An example of “a process of calculating a shielded area 60 a and anon-shielded area 60 b” by the analysis unit 130 will be described. Theanalysis unit 130 performs stereo matching for each portion of the twoimages included in the imaging information, and calculates a distancefrom the user 10 to the display 3 and a distance from the real object 11a to the display 3, which are included in the imaging information. Inthe following description, the distance between the user 10 and thedisplay 3 is referred to as a “first distance”. The distance between thereal object 11 a and the display 3 is referred to as a “seconddistance”.

The analysis unit 130 calculates the shielded area 60 a and thenon-shielded area 60 b from a geometric relationship between the size ofthe real object 11 a, the viewpoint positions of the user 10, the firstdistance, and the second distance. The shielded area 60 a indicates anarea that the user 10 cannot view due to the real object 11 a, of adisplay area of the display 3. The non-shielded area 60 b indicates anarea that the user 10 can view, of the display area of the display 3.

The analysis unit 130 calculates the size of the real object 11 a by theprocess of calculating the size of the real object 11 a. The process ofcalculating the viewpoint positions of the user 10 in the real space bythe analysis unit 130 is similar to those of the viewpoint positiondetection unit 110 and the display image generation unit 120.

The analysis unit 130 repeatedly performs the above processes every timethe imaging information is acquired from the image capturing unit 50,and outputs results of the analyses to the control unit 150. Inaddition, the analysis unit 130 may register some or all of the resultsof the analyses in an analysis result table of the storage unit 140.Furthermore, the analysis unit 130 may generate the model data 140 awhich is described later, on the basis of the images of the real object11 a included in the imaging information.

The storage unit 140 has the model data 140 a and the analysis resulttable 140 b. The storage unit 140 corresponds to a semiconductor memorydevice such as a random access memory (RAM) or flash memory, or astorage device such as a hard disk drive (HDD).

The model data 140 a is information (3D model) in which the shape,color, the reference size, and the like of each portion of the virtualobject 11 b are defined. The storage unit 140 may include model data inaddition to the model data about the virtual object 11 b, and thedisplay image generation unit 120 may select any of the model data.

The analysis result table 140 b is a table that holds the results of theanalyses by the analysis unit 130. FIG. 6 is a table illustrating anexample of a data structure of the analysis result table. As illustratedin FIG. 6 , the analysis result table 140 b contains the frame numbers,positions, viewpoint positions, sizes, attitudes, shielded areas, andnon-shielded areas.

In FIG. 6 , each of the frame numbers is a frame number assigned to aset of imaging information, and is information for uniquely identifyingthe imaging information. A record (position, viewpoint positions, size,attitude, shielded area, and non-shielded area) corresponding to thesame frame number is information obtained by analyzing the same imaginginformation.

In FIG. 6 , the positions each indicates a position (three-dimensionalcoordinates) of the real object 11 a in the real space. The respectiveviewpoint positions indicate viewpoint positions (coordinates of theleft eye and coordinates of the right eye) of the user 10 in the realspace. Each of the sizes indicates a size of the real object 11 a in thereal space, and is indicated by the vertical width H and the horizontalwidth W. Each of the attitudes indicates an attitude of the real object11 a, and is indicated by a difference θ between a position anddirection of each portion detected on the basis of the real object 11 aand the reference position and reference direction of each portion setin advance.

In FIG. 6 , each of the shielded areas corresponds to the shielded area60 a in FIG. 5 , and is indicated by coordinates of an upper leftposition and coordinates of a lower right position of the shielded area60 a. Each of the non-shielded area corresponds to the non-shielded area60 b in FIG. 5 , and is indicated by coordinates of an upper leftposition and coordinates of a lower right position of the non-shieldedarea 60 b.

The description returns to FIG. 4 . The control unit 150 is a processingunit that generates the control information on the basis of the resultsof the analyses by the analysis unit 130 and outputs the generatedcontrol information to the display image generation unit 120. It isassumed that a plurality of control policies are prepared in the controlunit 150 and the user 10 selects a desired control policy. Hereinafter,processes performed by the control unit 150 on the basis of controlpolicies P1 to P8 will be individually described, but a process in whicha plurality of policies are combined may be performed.

(Control Policy P1: Adjustment in Attitude)

In a case where the control policy P1 is selected, the control unit 150performs control to match the attitude of the real object 11 a with theattitude of the three-dimensional image (virtual object 11 b) displayedon the image display unit 60. For example, the control unit 150 acquiresthe attitude (θ) of the real object 11 a from the analysis unit 130,sets the acquired attitude in the control information, and outputs thecontrol information to the display image generation unit 120. Thedisplay image generation unit 120 adjusts the attitude set to the modeldata 140 a to match the attitude set in the control information, anduses the adjusted model data 140 a to cause the image display unit 60 todisplay the three-dimensional image. Furthermore, on the basis of theviewpoint positions of the user 10, the control unit 150 is configuredto determine the attitude of the virtual object 11 b so that theattitude of the real object 11 a and the attitude of the virtual object11 b look the same from the position of the user 10.

(Process Procedure Performed by Image Processing Apparatus Based onControl Policy P1)

FIG. 7 is a flowchart illustrating a process procedure performed by theimage processing apparatus based on the control policy P1. Asillustrated in FIG. 7 , the image processing apparatus 100 startsacquisition of the imaging information from the image capturing unit 50(Step S11). The viewpoint position detection unit 110 of the imageprocessing apparatus 100 detects each viewpoint position of the user 10(Step S12).

The analysis unit 130 of the image processing apparatus 100 detects thereal object 11 a held by the user 10 (Step S13). The analysis unit 130calculates the attitude (θ) of the real object 11 a (Step S14). Thecontrol unit 150 of the image processing apparatus 100 calculates theattitude of the virtual object 11 b according to the attitude of thereal object 11 a to generate the control information (Step S15).

The display image generation unit 120 of the image processing apparatus100 generates the information about the three-dimensional image on thebasis of the model data 140 a and the control information, and causesthe image display unit 60 to display the three-dimensional image (StepS16).

When the process is continued (Step S17, Yes), the image processingapparatus 100 proceeds to Step S12. On the other hand, when the processis not continued (Step S17, No), the image processing apparatus 100finishes the process.

The performance of the process by the image processing apparatus 100according to the control policy P1 makes it possible for the user 10 torefers to the three-dimensional image of the virtual object 11 b beingin the same attitude as the attitude of the real object 11 a held by theuser, thus, supporting the work of the user 10.

(Control Policy P2: Adjustment in Size)

In a case where the control policy P2 is selected, the control unit 150performs control to match the size of the real object 11 a with the sizeof the three-dimensional image (virtual object 11 b) displayed on theimage display unit 60. For example, the control unit 150 acquires thesize (H and W) of the real object 11 a from the analysis unit 130, setsthe acquired size in the control information, and outputs the controlinformation to the display image generation unit 120. The display imagegeneration unit 120 adjusts the size set to the model data 140 a tomatch the size set in the control information, and uses the adjustedmodel data 140 a to cause the image display unit 60 to display thethree-dimensional image.

FIG. 8 is a diagram illustrating processing performed by the controlunit based on the control policy P2. As illustrated in FIG. 8 , the sizeof a virtual object 11 b-1 is different from the size of the real object11 a. The control unit 150 resizes the virtual object 11 b-1 by theabove processing to have a virtual object 11 b-2, and the size of thevirtual object 11 b-1 and the size of the real object 11 a become thesame.

(Process Procedure Performed by Image Processing Apparatus Based onControl Policy P2)

FIG. 9 is a flowchart illustrating a process procedure performed by theimage processing apparatus based on the control policy P2. Asillustrated in FIG. 9 , the image processing apparatus 100 startsacquisition of the imaging information from the image capturing unit 50(Step S21). The viewpoint position detection unit 110 of the imageprocessing apparatus 100 detects each viewpoint position of the user 10(Step S22).

The analysis unit 130 of the image processing apparatus 100 detects thereal object 11 a held by the user 10 (Step S23). The analysis unit 130calculates the size (H and W) of the real object 11 a (Step S24). Thecontrol unit 150 of the image processing apparatus 100 calculates thesize of the virtual object according to the size of the real object 11 ato generate the control information (Step S25).

The display image generation unit 120 generates the information aboutthe three-dimensional image on the basis of the model data 140 a and thecontrol information, and causes the image display unit 60 to display theresized three-dimensional image (Step S26).

When the process is continued (Step S27, Yes), the image processingapparatus 100 proceeds to Step S22. On the other hand, when the processis not continued (Step S27, No), the image processing apparatus 100finishes the process.

The performance of the process by the image processing apparatus 100according to the control policy P2 makes it possible for the user 10 torefers to the three-dimensional image of the virtual object 11 b havingthe same size as that of the real object 11 a held by the user, thus,supporting the work of the user 10.

(Control Policy P3: Adjustment in Apparent Size)

The real object 11 a is positioned in front of the image display unit60, and therefore, the three-dimensional image looks relatively smallfor the user 10 even if the real object 11 a and the three-dimensionalimage (virtual object 11 b) have the same size. Therefore, the controlunit 150 performs control to adjust the relative sizes of the realobject 11 a and the three-dimensional image.

FIG. 10 is a diagram illustrating processing performed by the controlunit based on the control policy P3. The control unit 150 calculates aheight Hv of the virtual object 11 b on the basis of the results of theanalyses. In FIG. 10 , H_(r) indicates a height of the real object 11 a.D_(r) indicates a distance between the real object 11 a and the user 10,and corresponds to, for example, a distance obtained by subtracting thesecond distance from the first distance. Dv indicates a distance betweenthe user 10 and the virtual object 11 b (image display unit 60), andcorresponds to, for example, the first distance.

The control unit 150 calculates the height H_(v) of the virtual object11 b according to Formula (1). Although not illustrated, the controlunit 150 calculates a width W_(v) of the virtual object 11 b accordingto Formula (2). In Formula (2), W_(r) represents a width of the realobject 11 a. In the following description, the height H_(v) and thewidth W_(v) are collectively referred to as a “size after resizing”.

H _(v) =H _(r) ×D _(v) /D _(r)  (1)

W _(v) =W _(r) ×D _(v) /D _(r)  (2)

The control unit 150 calculates the size after resizing by the aboveprocessing, sets the size after resizing in the control information, andoutputs the control information to the display image generation unit120. The display image generation unit 120 adjusts the size set in themodel data 140 a to match the size after resizing set in the controlinformation, and uses the adjusted model data 140 a to cause the imagedisplay unit 60 to display the three-dimensional image.

(Process Procedure Performed by Image Processing Apparatus Based onControl Policy P3)

FIG. 11 is a flowchart illustrating a process procedure performed by theimage processing apparatus based on the control policy P3. Asillustrated in FIG. 11 , the image processing apparatus 100 startsacquisition of the imaging information from the image capturing unit 50(Step S31). The viewpoint position detection unit 110 of the imageprocessing apparatus 100 detects each viewpoint position of the user 10(Step S32).

The analysis unit 130 of the image processing apparatus 100 detects thereal object 11 a held by the user 10 (Step S33). The analysis unit 130calculates the size (H and W) of the real object 11 a (Step S34). Thecontrol unit 150 of the image processing apparatus 100 calculates thesize (H_(v), W_(v)) after resizing and generates the control information(Step S35).

The display image generation unit 120 generates the information aboutthe three-dimensional image on the basis of the model data 140 a and thecontrol information, and causes the image display unit 60 to display theresized three-dimensional image (Step S36).

When the process is continued (Step S37, Yes), the image processingapparatus 100 proceeds to Step S32. On the other hand, when the processis not continued (Step S37, No), the image processing apparatus 100finishes the process.

Even if the real object 11 a is positioned closer to the user 10 than tothe image display unit 60, the performance of the process by the imageprocessing apparatus 100 according to the control policy P3 makes itpossible for the user 10 to refers to the three-dimensional image of thevirtual object 11 b having an apparent size the same as the size of thereal object 11 a held by the user, thus, supporting the work of the user10.

(Control Policy P4: Movement to Position Free from Overlapping)

When the three-dimensional image of the virtual object 11 b is displayedin the shielded area of the image display unit 60, the real object 11 aoverlaps the three-dimensional image. Therefore, the control unit 150performs control to cause the three-dimensional image to be displayed inthe non-shielded area of the image display unit 60.

FIG. 12 is a diagram illustrating processing performed by the controlunit based on the control policy P4. As described above, the analysisunit 130 analyzes the imaging information to calculate the shielded area60 a and the non-shielded area 60 b. The control unit 150 adjusts adisplay position of the three-dimensional image according to any one ofa control policy P4-1, a control policy P4-2, and a control policy P4-3.The display position of the three-dimensional image before adjusting isdenoted by “R”. The display position of the three-dimensional imageafter adjusting is denoted by “Q”.

The “control policy P4-1” will be described. The control unit 150calculates a display position Q1 of the three-dimensional image at whichthe entire virtual object 11 b is included in the non-shielded area 60 bincluded in a result of the analysis. For example, the control unit 150calculates the display position Q1 so that a distance from the positionR to the display position Q1 is minimized.

The “control policy P4-2” will be described. The control unit 150calculates a display position Q2 in the non-shielded area 60 b so that adistance from the position R to the display position Q2 is maximized.

The “control policy P4-3” will be described. The control unit 150calculates a display position Q3 that is closer to the center portion ofthe image display unit 60 (display 3). This is because the image qualityof the three-dimensional image is improved toward the center of thedisplay 3. For example, the control unit 150 calculates the displayposition Q3 on the basis of Formula (3).

C=min(C _(RQ)(D _(RQ))+C _(CQ)(D _(CQ)))  (3)

In Formula (3), C_(RQ) (D_(RQ)) is a function with which the costdecreases as the display position Q3 determined by D_(RQ) is fartherfrom the shielded area 60 a. C_(CQ) (D_(CQ)) is a function indicatingthat a cost decreases as the display position Q3 determined by D_(CQ) iscloser to the center of display 3. The control unit 150 calculates aposition where the value of Formula (3) is minimized as the finaldisplay position Q3.

The control unit 150 calculates a display position of the virtual objectlib on the basis of any one of the control policies P4-1 to P4-3. Thecontrol unit 150 sets the calculated display position in the controlinformation and outputs the control information to the display imagegeneration unit 120. The display image generation unit 120 adjusts thedisplay position of the three-dimensional image to the display positionset in the control information and causes the image display unit 60 todisplay the three-dimensional image.

(Process Procedure Performed by Image Processing Apparatus Based onControl Policy P4)

FIG. 13 is a flowchart illustrating a process procedure performed by theimage processing apparatus based on the control policy P4. Asillustrated in FIG. 13 , the image processing apparatus 100 startsacquisition of the imaging information from the image capturing unit 50(Step S41). The viewpoint position detection unit 110 of the imageprocessing apparatus 100 detects each viewpoint position of the user 10(Step S42).

The analysis unit 130 of the image processing apparatus 100 detects thereal object 11 a held by the user 10 (Step S43). The analysis unit 130calculates a positional relationship between the real object 11 a andthe virtual object 11 b (Step S44). The control unit 150 of the imageprocessing apparatus 100 identifies the shielded area and thenon-shielded area (Step S45).

The control unit 150 calculates the display position included in thenon-shielded area and generates the control information (Step S46). Thedisplay image generation unit 120 generates the information about thethree-dimensional image on the basis of the model data 140 a and thecontrol information, and causes the image display unit 60 to display thethree-dimensional image in the non-shielded area (Step S47).

When the process is continued (Step S48, Yes), the image processingapparatus 100 proceeds to Step S42. On the other hand, when the processis not continued (Step S48, No), the image processing apparatus 100finishes the process.

The performance of the process by the image processing apparatus 100according to the control policy P4 prevents overlapping of the realobject 11 a and the three-dimensional image (virtual object 11 b),enables the user 10 to simultaneously refer to both the real object 11 aand the three-dimensional image, supporting the work of the user 10.

(Control Policy P5: Automatic Selection of Overlapping/Non-Overlapping)

The work of the user 10 sometimes includes “viewing the real object 11 aand the three-dimensional image in an overlapping manner” and “viewingthe real object 11 a and the three-dimensional image in an alignedmanner”. Therefore, the control unit 150 controls the display positionof the three-dimensional image on the basis of a distance D between anapparent position of the real object 11 a and the display position ofthe three-dimensional image. Here, in an example, the apparent positionof the real object 11 a is defined as a position where a straight linepassing through the viewpoint position of the user 10 and the positionof the real object 11 a intersect the display 3.

FIG. 14 is a diagram (1) illustrating processing performed by thecontrol unit based on the control policy P5. As illustrated in FIG. 14 ,when a distance from the apparent position of the real object 11 a tothe virtual object 11 b (three-dimensional image) is less than athreshold, the control unit 150 causes the virtual object 11 b to bedisplayed in the shielded area. On the other hand, when the distancefrom the apparent position of the real object 11 a to the virtual object11 b (three-dimensional image) is equal to or larger than the threshold,the control unit 150 causes the virtual object 11 b to be displayed inthe non-shielded area.

FIG. 15 is a diagram (2) illustrating processing performed by thecontrol unit based on the control policy P5. In FIG. 15 , the apparentposition of the real object 11 a is denoted by “S”. The control unit 150calculates the apparent position “S” on the basis of a relationshipbetween the viewpoint position of the user 10, the position of the realobject 11 a in the space, and the position of the display 3. Theposition “Q” indicates the display position of the three-dimensionalimage.

The control unit 150 calculates the distance D between the position Sand the position Q, and sets the display position of thethree-dimensional image to the position S when the distance D is lessthan a threshold D_(th). On the other hand, when the control unit 150calculates the distance D between the position S and the position Q, andsets the display position of the display image so as to be in thenon-shielded area 60 b when the distance D is equal to or larger thanthe threshold D_(th).

The control unit 150 calculates the display position of thethree-dimensional image by the above processing, sets the displayposition in the control information, and outputs the control informationto the display image generation unit 120. The display image generationunit 120 adjusts the display position of the three-dimensional image tothe display position set in the control information and causes the imagedisplay unit 60 to display the three-dimensional image.

(Process Procedure Performed by Image Processing Apparatus Based onControl Policy P5)

FIG. 16 is a flowchart illustrating a process procedure performed by theimage processing apparatus based on the control policy P5. Asillustrated in FIG. 16 , the image processing apparatus 100 startsacquisition of the imaging information from the image capturing unit 50(Step S51). The viewpoint position detection unit 110 of the imageprocessing apparatus 100 detects each viewpoint position of the user 10(Step S52).

The analysis unit 130 of the image processing apparatus 100 detects thereal object 11 a held by the user 10 (Step S53). The analysis unit 130calculates the positional relationship between the real object 11 a andthe virtual object 11 b (Step S54). The control unit 150 of the imageprocessing apparatus 100 identifies the shielded area and thenon-shielded area (Step S55).

The control unit 150 calculates the distance D between the apparentposition of the real object 11 a and the position of the virtual object11 b (Step S56). The control unit 150 determines whether the distance Dis less than the threshold D_(t)h (Step S57).

When the distance D is less than the threshold D_(th) (Step S57, Yes),the control unit 150 calculates the display position included in theshielded area, generates the control information (Step S58), andproceeds to Step S60.

On the other hand, when the distance D is not less than the thresholdD_(th) (Step S57, No), the control unit 150 calculates the displayposition included in the non-shielded area, generates the controlinformation (Step S59), and proceeds to Step S60.

The display image generation unit 120 generates the information aboutthe three-dimensional image on the basis of the model data 140 a and thecontrol information, and causes the image display unit 60 to display thethree-dimensional image at the display position (Step S60). When theprocess is continued (Step S60, Yes), the image processing apparatus 100proceeds to Step S52. On the other hand, when the process is notcontinued (Step S48, No), the image processing apparatus 100 finishesthe process.

The image processing apparatus 100 performs the process according to thecontrol policy P5, and thus, the control unit 150 causes the virtualobject 11 b to be displayed in the shielded area when the distance fromthe apparent position of the real object 11 a to the virtual object 11 b(three-dimensional image) is less than the threshold. On the other hand,when the distance from the apparent position of the real object 11 a tothe virtual object 11 b (three-dimensional image) is equal to or largerthan the threshold, the control unit 150 causes the virtual object 11 bto be displayed in the non-shielded area. This configuration makes itpossible to support the work of the user 10, both for the “viewing thereal object 11 a and the three-dimensional image in an overlappingmanner” and the “viewing the real object 11 a and the three-dimensionalimage in an aligned manner”.

(Control Policy P6: Motion Parallax)

For example, the user 10 fixes the viewpoint positions and moves andviews the real object 11 a. The control unit 150 shows a changecorresponding to movement of the real object 11 a by moving the displayposition of the virtual object 11 b instead.

FIG. 17 is a diagram (1) illustrating processing performed by thecontrol unit based on the control policy P6. As illustrated in FIG. 17 ,it is assumed that the user 10 moves the real object 11 a in a “rightdirection”. In this case, the control unit 150 moves the virtual object11 b in the “right direction” to show the change corresponding to themovement of the real object 11 a by moving the display position of thevirtual object instead.

FIG. 18 is a diagram (2) illustrating processing performed by thecontrol unit based on the control policy P6. As illustrated in FIG. 18 ,the user 10 moves the real object 11 a to the right. If the viewpointposition of the user 10 does not move, the movement of the real object11 a to the right makes it possible to refer to a left side surface ofthe real object 11 a. The control unit 150 moves the display position ofthe virtual object 11 b from a position R1 to a position R2, andtherefore, the user 10 is allowed to refer to the left side surface ofthe virtual object 11 b. In other words, the change corresponding to themovement of the real object 11 a is shown by moving the display positionof the virtual object 11 b instead.

The control unit 150 calculates a movement trajectory of the real object11 a on the basis of the positions of the real object included inresults of the analyses for the respective imaging information, andcalculates the movement of the virtual object 11 b. For example, thecontrol unit 150 transforms the movement trajectory in a predeterminedperiod to a coordinate system of the image display unit 60 to calculatethe movement of the virtual object 11 b. Motion information about thevirtual object 11 b includes information in which a plurality ofcoordinates on the image display unit 60 is arranged in chronologicalorder.

The control unit 150 calculates the movement of the virtual object 11 bby the above processing, sets the movement of the virtual object 11 b inthe control information, and outputs the control information to thedisplay image generation unit 120. The display image generation unit 120moves the three-dimensional image displayed on the image display unit60, on the basis of the movement set in the control information.

(Process Procedure Performed by Image Processing Apparatus Based onControl Policy P6)

FIG. 19 is a flowchart illustrating a process procedure performed by theimage processing apparatus based on the control policy P6. Asillustrated in FIG. 19 , the image processing apparatus 100 startsacquisition of the imaging information from the image capturing unit 50(Step S71). The analysis unit 130 of the image processing apparatus 100detects the real object 11 a held by the user 10 (Step S72).

The control unit 150 of the image processing apparatus 100 calculatesthe movement trajectory of the real object 11 a on the basis of resultsof the analyses (Step S73). The control unit 150 calculates the movementof the virtual object 11 b and generates the control information (StepS74).

The display image generation unit 120 generates the information aboutthe three-dimensional image on the basis of the model data 140 a and thecontrol information, and causes the image display unit 60 to display andmove the three-dimensional image (Step S75).

When the process is continued (Step S76, Yes), the image processingapparatus 100 proceeds to Step S72. On the other hand, when the processis not continued (Step S76, No), the image processing apparatus 100finishes the process.

The performance of the process by the image processing apparatus 100according to the control policy P6 makes it possible to show the changecorresponding to the movement of the real object 11 a by moving thedisplay position of the virtual object 11 b instead, supporting the workof the user 10.

(Control Policy P7: Recording/Reproducing Movement)

For example, as described above, the analysis unit 130 records theresults (the position of the real object 11 a) of the analyses of therespective imaging information in the analysis result table 140 b inchronological order. The control unit 150 calculates the movement of thereal object 11 a on the basis of the positions of the real object 11 ain a certain period, recorded in the analysis result table 140 b. Thecontrol unit 150 calculates the movement of the virtual object 11 b onthe basis of the movement of the real object 11 a, and moves the virtualobject 11 b on the basis of this movement, whereby the movement of thereal object 11 a in the past can be reproduced.

Here, in a case where the movement of the real object 11 a is used, theanalysis unit 130 may calculate moving average of the real object 11 ato remove noise. In addition, the analysis unit 130 may record differentmovements of the real object 11 a and combine the recorded positions toedit the different short movements into a series of motion information.For example, the analysis unit 130 may combine the movement trajectoryduring a first period, a second period, and a third period.

FIG. 20 is a diagram illustrating processing performed by the controlunit based on the control policy P7. The user 10 moves the real object11 a for the certain period, and the analysis unit 130 records themovement of the real object 11 a (positions indicated by the framenumbers) in the analysis result table 140 b on the basis of the imaginginformation. The control unit 150 calculates the movement of the virtualobject 11 b on the basis of the movement of the real object 11 a in thecertain period, recorded in the analysis result table 140 b, andgenerates the control information for moving the virtual object 11 b onthe basis of this movement. For example, the control unit 150 performscalculation to transform the position (position in the real space) ofeach frame that forms the movement of the real object 11 a to thecoordinate system of the display screen of an image display unit 61. Itis assumed that a relationship between the position in the real spaceand the coordinates of the display screen is defined in advance.

(Process Procedure Performed by Image Processing Apparatus Based onControl Policy P7)

FIG. 21 is a flowchart illustrating a process procedure performed by theimage processing apparatus based on the control policy P7. Asillustrated in FIG. 21 , the image processing apparatus 100 startsacquisition of the imaging information from the image capturing unit 50(Step S81). The analysis unit 130 of the image processing apparatus 100detects the real object 11 a held by the user 10 (Step S82).

The analysis unit 130 calculates the position of the real object 11 aand records the position in the analysis result table 140 b (Step S83).When the certain period has not elapsed (Step S84, No), the imageprocessing apparatus 100 proceeds to Step S82. On the other hand, whenthe certain period has elapsed (Step S84, Yes), the image processingapparatus 100 proceeds to Step S85.

The control unit 150 of the image processing apparatus 100 calculatesthe movement of the virtual object 11 b on the basis of each position ofthe real object 11 a recorded in the analysis result table 140 b, andgenerates the control information (Step S85). The display imagegeneration unit 120 generates the information about thethree-dimensional image on the basis of the model data 140 a and thecontrol information, and causes the image display unit 60 to display andmove the three-dimensional image (Step S86).

The performance of the process by the image processing apparatus 100according to the control policy P7 to record the movement of the realobject 11 a for a certain period, calculates the movement of the virtualobject 11 b on the basis of the recorded motion, and controls themovement of the three-dimensional image. This configuration makes itpossible to reproduce the movement of the real object 11 a in the pastwith the virtual object 11 b.

(Control Policy P8: Illumination)

In a case where specification of illumination position and illuminationcolor is received from outside, the control unit 150 calculates thecolor of and a reflection position on the virtual object 11 b on thebasis of the illumination position and illumination color, and generatesthe control information. The display image generation unit 120 generatesthe information about the three-dimensional image on the basis of thecolor of and the reflection position on the virtual object 11 b set inthe control information, and causes the image display unit 60 to displaythe information.

FIG. 22 is a diagram illustrating processing performed by the controlunit based on the control policy P8. The user 10 specifies theillumination position of an illumination lamp 12 and the illuminationcolor, to the image processing apparatus 100, with an input device (notillustrated) or the like. Furthermore, the analysis unit 130 mayestimate the illumination position and the illumination color by usingthe imaging information and output information about the analyzedillumination position and illumination color to the control unit 150.The control unit 150 calculates the color of and the reflection positionon the virtual object 11 b-1 on the basis of the illumination positionand the illumination color. The display image generation unit 120performs correction to add effects (color and gloss) to the virtualobject 11 b-1 on the basis of the color and reflection positioncalculated by the analysis unit 130, generates the virtual object 11b-2, and causes the image display unit 60 to display the virtual object11 b-2.

When the color of the real object 11 a changes, the control unit 150changes the color of the virtual object 11 b-2. Furthermore, when theillumination position changes, the control unit 150 may change theposition of specular reflection on the virtual object 11 b-2.

(Process Procedure Performed by Image Processing Apparatus Based onControl Policy P8)

FIG. 23 is a flowchart illustrating a process procedure performed by theimage processing apparatus based on the control policy P8. Asillustrated in FIG. 23 , the image processing apparatus 100 startsacquisition of the imaging information from the image capturing unit 50(Step S91). The viewpoint position detection unit 110 of the imageprocessing apparatus 100 detects each viewpoint position of the user 10(Step S92).

The analysis unit 130 of the image processing apparatus 100 detects thereal object 11 a held by the user 10 (Step S93). The control unit 150 ofthe image processing apparatus 100 receives the specification of theillumination color and the illumination position (or the analysis unit130 estimates the illumination position and the illumination color byusing the imaging information, and outputs the information about theanalyzed illumination position and illumination color to the controlunit 150) (Step S94). The control unit 150 calculates the color of andthe reflection position on the virtual object 11 b according to theillumination color and the illumination position, and generates thecontrol information (Step S95).

The display image generation unit 120 generates the information aboutthe three-dimensional image on the basis of the model data 140 a and thecontrol information, and displays the three-dimensional image on whichthe effects of the illumination are reflected (Step S96).

When the process is continued (Step S97, Yes), the image processingapparatus 100 proceeds to Step S92. On the other hand, when the processis not continued (Step S97, No), the image processing apparatus 100finishes the process.

The performance of the process by the image processing apparatus 100according to the control policy P8 makes it possible to reflect theeffects of the illumination similar to those in illuminating the realobject 11 a, in the virtual object 11 b.

«1.4. Effects of Image Processing Apparatus According to FirstEmbodiment»

As described above, in the image processing apparatus according to thefirst embodiment, the analysis unit 130 analyzes the imaginginformation, and the control unit 150 generates the control informationbased on the control policies P1 to P8, on the basis of the results ofthe analyses and controls the display of the three-dimensional image,and therefore, the three-dimensional image can be appropriatelydisplayed for the user. For example, in a case where the real object 11a having a 3D shape is displayed as the virtual object 11 b, on theimage display unit 60, the above processing makes it possible toautomatically assist the work of viewing and comparing the real object11 a and the virtual object 11 b simultaneously can be automatically,improving working efficiency for the user.

2. Second Embodiment «2.1. Situation According to Second Embodiment»

FIG. 24 is a diagram illustrating an example of a situation according toa second embodiment. As illustrated in FIG. 24 , an image processingapparatus 200 and an image processing apparatus 300 are connected toeach other via a network 20.

It is assumed that the user 10 is in front of the image processingapparatus 200, and the user 10 holds the real object 11 a. The imageprocessing apparatus 200 images the real object 11 a by a camera 4 a andcalculates the movement of the real object 11 a. The image processingapparatus 200 sets motion information about the real object 11 a in thecontrol information and transmits the control information to the imageprocessing apparatus 300. The image processing apparatus 200 may displaythe three-dimensional image of the virtual object 11 b on a display 3 aas in the image processing apparatus 100 according to the firstembodiment.

Note that the image processing apparatus 300 may perform processingsimilar to that of the image processing apparatus 100 by using a camera4 b, separately from the above processing.

When receiving the control information, the image processing apparatus300 displays the three-dimensional image of the virtual object 11 b on adisplay 3 b, on the basis of the model data about the virtual object 11b and the control information. The motion information about the realobject 11 a is set in the control information, and the image processingapparatus 300 controls the movement of the virtual object 11 b accordingto the movement of the real object 11 a.

«2.2. Configuration of Image Processing Apparatus According to SecondEmbodiment

FIG. 25 is a diagram (1) illustrating a configuration example of theimage processing apparatus according to the second embodiment. Asillustrated in FIG. 25 , the image processing apparatus 200 is connectedto an image capturing unit 51 and an image display unit 61. Note thatthe image processing apparatus 200 may include the image capturing unit51 and the image display unit 61.

The image capturing unit 51 is a stereo camera imaging the user 10 andthe real object 11 a held by the user 10 that are within the imagingrange. The image capturing unit 51 corresponds to the camera 4 aillustrated in FIG. 24 . The image capturing unit 51 includes anotherimage sensor such as a CMOS sensor or CCD. The image capturing unit 51outputs information about an image captured to the image processingapparatus 200. In the following description, the information about theimage captured by the image capturing unit 51 is referred to as the“imaging information”.

Since the image capturing unit 51 is the stereo camera, the imaginginformation includes an image captured by one of the cameras and animage captured by the other camera. The image capturing unit 51 capturesan image at a predetermined frame rate, and outputs the imaginginformation to the image processing apparatus 200 every time an image iscaptured. Frame numbers may be assigned to the imaging information inchronological order in ascending order.

The image display unit 61 is a display device that displays athree-dimensional image on the basis of information output from theimage processing apparatus 200. The image display unit 61 corresponds tothe display 3 a illustrated in FIG. 24 . For example, the image displayunit 61 displays the three-dimensional image of the virtual object 11 b.

The image processing apparatus 200 includes a viewpoint positiondetection unit 210, a display image generation unit 220, an analysisunit 230, a storage unit 240, a control unit 250, and a communicationunit 260.

The viewpoint position detection unit 210 is a processing unit thatperforms face detection on the basis of the imaging information anddetects the viewpoint positions of the user 10. The viewpoint positiondetection unit 210 outputs viewpoint position information to the displayimage generation unit 220. Other description of the viewpoint positiondetection unit 210 are similar to those of the viewpoint positiondetection unit 110 described in the first embodiment.

The display image generation unit 220 is a processing unit thatgenerates information for causing the image display unit 61 to displaythe three-dimensional image and outputs the generated information to theimage display unit 61. Other description of the display image generationunit 220 are similar to those of the display image generation unit 120described in the first embodiment.

The analysis unit 230 is a processing unit that performs variousanalyses on the basis of the imaging information acquired from the imagecapturing unit 51 and outputs results of the analyses to the controlunit 250. Other processing of the analysis unit 230 is similar to thatof the analysis unit 130 described in the first embodiment.

The storage unit 240 has model data 240 a and an analysis result table240 b. The storage unit 240 corresponds to a semiconductor memory devicesuch as RAM or a flash memory, or a storage device such as HDD. Thedescription of the model data 240 a is similar to the description of themodel data 140 a made in the first embodiment. The description of theanalysis result table 240 b is similar to the description of theanalysis result table 140 b made in the first embodiment.

The control unit 250 is a processing unit that generates the controlinformation on the basis of the results of the analyses by the analysisunit 230 and outputs the generated control information to the displayimage generation unit 220. Furthermore, the control unit 250 transmitsthe control information to the image processing apparatus 300 via thecommunication unit 260.

For example, the control unit 250 performs calculation to transform theposition of the real object 11 a included in the analysis result to acoordinate system of a display screen of an image display unit 62 of theimage processing apparatus 300 which is described later, setsinformation about the calculated position in the control information,and transmits the control information to the image processing apparatus300.

The communication unit 260 is a processing unit that transmits thecontrol information acquired from the control unit 250 to the imageprocessing apparatus 300 via the network 20. The communication unit 260corresponds to a communication device such as a network interface card(NIC).

FIG. 26 is a diagram (2) illustrating a configuration example of animage processing apparatus according to the second embodiment. Asillustrated in FIG. 26 , the image processing apparatus 300 is connectedto an image capturing unit 52 and the image display unit 62. Note thatthe image processing apparatus 300 may include the image capturing unit52 and the image display unit 62.

The image capturing unit 52 is a stereo camera that images a user 15 whois within the imaging range. The image capturing unit 52 corresponds tothe camera 4 b illustrated in FIG. 24 . The image capturing unit 52includes another image sensor such as a CMOS sensor or CCD. The imagecapturing unit 52 outputs information about an image captured to theimage processing apparatus 200. In the following description, theinformation about the image captured by the image capturing unit 52 isreferred to as the “imaging information”.

The image display unit 62 is a display device that displays athree-dimensional image on the basis of information output from theimage processing apparatus 300. The image display unit 62 corresponds tothe display 3 b illustrated in FIG. 24 . For example, the image displayunit 62 displays the three-dimensional image of the virtual object 11 b.

The image processing apparatus 300 includes a viewpoint positiondetection unit 310, a display image generation unit 320, an analysisunit 330, a storage unit 340, a control unit 350, and a communicationunit 360. Here, in the image processing apparatus 300, a “firstoperation mode” or a “second operation mode” is set in advance. Thefirst operation mode is a mode in which the image processing apparatus300 receives the control information from the image processing apparatus200 via the network 20 and displays the three-dimensional image of thevirtual object 11 b on the basis of the received control information.

The second operation mode is a mode in which the image processingapparatus 300 analyzes the imaging information captured by itself byusing the camera 4 b and displays the three-dimensional image of thevirtual object 11 b. Processing performed by the viewpoint positiondetection unit 310, the display image generation unit 320, the analysisunit 330, and the control unit 350 of the image processing apparatus 300in the second operation mode is similar to the processing performed bythe viewpoint position detection unit 110, the display image generationunit 120, the analysis unit 130, and the control unit 150 of the imageprocessing apparatus 100 which has been described in the firstembodiment, and the description thereof will be omitted.

Hereinafter, processing performed by the viewpoint position detectionunit 310, the display image generation unit 320, the analysis unit 330,the control unit 350, and the communication unit 360 of the imageprocessing apparatus 300 in the first operation mode will be described.

The viewpoint position detection unit 310 is a processing unit thatperforms face detection on the basis of the imaging information anddetects the viewpoint positions of the user 15. The viewpoint positiondetection unit 210 outputs viewpoint position information to the displayimage generation unit 320. Other description of the viewpoint positiondetection unit 310 are similar to those of the viewpoint positiondetection unit 110 described in the first embodiment.

The display image generation unit 320 is a processing unit thatgenerates information for causing the image display unit 62 to displaythe three-dimensional image and outputs the generated information to theimage display unit 62. Other description of the display image generationunit 320 are similar to those of the display image generation unit 120described in the first embodiment.

In the first operation mode, the analysis unit 330 may pause theprocessing, or may perform processing similar to that of the analysisunit 130 described in the first embodiment.

The storage unit 340 has model data 340 a and an analysis result table340 b. The storage unit 340 corresponds to a semiconductor memory devicesuch as RAM or a flash memory, or a storage device such as HDD. Thedescription of the model data 340 a is similar to the description of themodel data 140 a made in the first embodiment. The description of theanalysis result table 340 b is similar to the description of theanalysis result table 140 b made in the first embodiment.

The control unit 250 is a processing unit that acquires the controlinformation from the image processing apparatus 200 via thecommunication unit 360 and outputs the acquired control information tothe display image generation unit 320.

The communication unit 360 is a processing unit that receives thecontrol information from the image processing apparatus 200 via thenetwork 20. The communication unit 360 corresponds to a communicationdevice such as NIC.

(Process Procedure Performed by Image Processing Apparatus 200)

FIG. 27 is a flowchart (1) illustrating a process procedure performed bythe image processing apparatus according to the second embodiment. Asillustrated in FIG. 27 , the image processing apparatus 200 startsacquisition of the imaging information from the image capturing unit 51(Step S100). The viewpoint position detection unit 210 of the imageprocessing apparatus 200 detects each viewpoint position of the user 10(Step S101).

The analysis unit 230 of the image processing apparatus 200 detects thereal object 11 a held by the user 10 (Step S102). The control unit 250of the image processing apparatus 200 calculates the movement of thereal object 11 a (Step S103). The control unit 250 of the imageprocessing apparatus 200 calculates the movement of the virtual object11 b according to the movement of the real object 11 a to generate thecontrol information (Step S104).

The communication unit 260 of the image processing apparatus 200transmits the control information to the image processing apparatus 300(Step S105). When the process is continued (Step S106, Yes), the imageprocessing apparatus 200 proceeds to Step S101. On the other hand, whenthe process is not continued (Step S106, No), the image processingapparatus 200 finishes the process.

(Process Procedure Performed by Image Processing Apparatus 300)

FIG. 28 is a flowchart (2) illustrating a process procedure performed bythe image processing apparatus according to the second embodiment. Asillustrated in FIG. 28 , the image processing apparatus 300 startsacquisition of the imaging information from the image capturing unit 52(Step S150). The viewpoint position detection unit 310 of the imageprocessing apparatus 300 detects each viewpoint position of the user 15(Step S151).

The communication unit 360 of the image processing apparatus 300receives the control information from the image processing apparatus 200(Step S152). The control unit 350 of the image processing apparatus 300outputs the control information to the display image generation unit 320(Step S153).

The display image generation unit 320 of the image processing apparatus300 generates the information about the three-dimensional image on thebasis of the model data 340 a and the control information, and causesthe image display unit 60 to display and move the three-dimensionalimage (Step S154).

When the process is continued (Step S155, Yes), the image processingapparatus 300 proceeds to Step S151. On the other hand, when the processis not continued (Step S155, No), the image processing apparatus 300finishes the process.

«2.3. Effects of Image Processing Apparatus According to SecondEmbodiment»

As described above, in the image processing apparatus 200, the analysisunit 230 analyzes the imaging information, and the control unit 250generates the control information on the basis of the results of theanalyses and transmits the control information to the image processingapparatus 300. When receiving the control information, the imageprocessing apparatus 300 displays the three-dimensional image of thevirtual object 11 b on the display 3 b, on the basis of the model data340 a and the control information. This configuration makes it possibleto reproduce the movement of the real object 11 a held by the user 10positioned at a place different from a place where the user 15 ispositioned, with the three-dimensional image displayed by the imageprocessing apparatus 300.

3. Third Embodiment «3.1. Example of Appearance of Image ProcessingApparatus»

FIG. 29 is a diagram illustrating an example of a situation according toa third embodiment. As illustrated in FIG. 29 , an image processingapparatus 400 and a terminal device 70 are connected to each other viathe network 20. The terminal device 70 corresponds to a tablet terminal,a smartphone, or the like.

The user 10 holds the terminal device 70. When the user 10 moves theterminal device 70, the terminal device 70 calculates motion informationabout the terminal device 70 by using an acceleration sensor or thelike, and transmits the control information in which the motioninformation is set, to the image processing apparatus 400. Whenreceiving the control information, the image processing apparatus 400displays the three-dimensional image of the virtual object 11 b on thedisplay 3, on the basis of the model data about the virtual object 11 band the control information.

«3.2. Configuration of Image Processing Apparatus According to ThirdEmbodiment»

FIG. 30 is a diagram illustrating a configuration example of the imageprocessing apparatus according to the third embodiment. As illustratedin FIG. 30 , the image processing apparatus 400 is connected to an imagecapturing unit 53 and an image display unit 63. Note that the imageprocessing apparatus 400 may include the image capturing unit 53 and theimage display unit 63.

The image capturing unit 53 is a stereo camera that images the user 15who is within an imaging range. The image capturing unit 51 correspondsto the camera 4 illustrated in FIG. 29 . The image capturing unit 53includes another image sensor such as a CMOS sensor or CCD. The imagecapturing unit 53 outputs information about an image captured to theimage processing apparatus 400. In the following description, theinformation about the image captured by the image capturing unit 53 isreferred to as the “imaging information”.

Since the image capturing unit 53 is the stereo camera, the imaginginformation includes an image captured by one of the cameras and animage captured by the other camera. The image capturing unit 53 capturesan image at a predetermined frame rate, and outputs the imaginginformation to the image processing apparatus 400 every time an image iscaptured. Frame numbers may be assigned to the imaging information inchronological order in ascending order.

The image display unit 63 is a display device that displays athree-dimensional image on the basis of information output from theimage processing apparatus 400. The image display unit 63 corresponds tothe display 3 illustrated in FIG. 29 . For example, the image displayunit 63 displays the three-dimensional image of the virtual object 11 b.

The image processing apparatus 400 includes a viewpoint positiondetection unit 410, a display image generation unit 420, an analysisunit 430, a storage unit 440, a control unit 450, and a communicationunit 460.

The viewpoint position detection unit 410 is a processing unit thatperforms face detection on the basis of the imaging information anddetects the viewpoint positions of the user 15. The viewpoint positiondetection unit 410 outputs viewpoint position information to the displayimage generation unit 420. Other description of the viewpoint positiondetection unit 410 are similar to those of the viewpoint positiondetection unit 110 described in the first embodiment.

The display image generation unit 420 is a processing unit thatgenerates information for causing the image display unit 63 to displaythe three-dimensional image and outputs the generated information to theimage display unit 63. Other description of the display image generationunit 420 are similar to those of the display image generation unit 120described in the first embodiment.

The analysis unit 430 is a processing unit that performs variousanalyses on the basis of the imaging information acquired from the imagecapturing unit 53 and outputs results of the analyses to the controlunit 450. Other processing of the analysis unit 430 is similar to thatof the analysis unit 130 described in the first embodiment. Note that,in a case where the real object 11 a is not included in the imaginginformation, the analysis unit 430 does not perform the processing ofdetecting the real object 11 a.

The storage unit 440 has model data 440 a and an analysis result table440 b. The storage unit 440 corresponds to a semiconductor memory devicesuch as RAM or a flash memory, or a storage device such as HDD. Thedescription of the model data 440 a is similar to the description of themodel data 140 a made in the first embodiment. The description of theanalysis result table 440 b is similar to the description of theanalysis result table 140 b made in the first embodiment.

The control unit 450 is a processing unit that acquires the controlinformation from the terminal device 70 and outputs the acquired controlinformation to the display image generation unit 420. Note that thecontrol unit 450 may transform a change in the movement included in thecontrol information acquired from the terminal device 70 to thecoordinate system of the image display unit 60 to output, to the displayimage generation unit 420, the control information updated so as tomatch the movement of the virtual object 11 b.

The communication unit 460 is a processing unit that receives thecontrol information from the terminal device 70 via the network 20. Thecommunication unit 460 corresponds to a communication device such asNIC.

(Process Procedure Performed by Terminal Device 70)

FIG. 31 is a flowchart illustrating a process procedure performed by theterminal device according to the third embodiment. As illustrated inFIG. 31 , the terminal device 70 starts detection of the movement byusing the acceleration sensor or the like set in the terminal deviceitself (Step S200). The terminal device 70 sets the motion informationabout the terminal device 70 in the control information (Step S201).

The terminal device 70 transmits the control information to the imageprocessing apparatus 400 (Step S202). When the process is continued(Step S203, Yes), the terminal device 70 proceeds to Step S201. When theprocess is not continued (Step S203, No), the terminal device 70finishes the process.

(Process Procedure Performed by Image Processing Apparatus 400)

FIG. 32 is a flowchart illustrating a process procedure performed by theimage processing apparatus according to the third embodiment. Asillustrated in FIG. 32 , the image processing apparatus 400 startsacquisition of the imaging information from the image capturing unit 53(Step S250). The viewpoint position detection unit 410 of the imageprocessing apparatus 400 detects each viewpoint position of the user 15(Step S251).

The communication unit 460 of the image processing apparatus 400receives the control information from the terminal device 70 (StepS252). The control unit 450 of the image processing apparatus 400outputs the control information to the display image generation unit 420(Step S253).

The display image generation unit 420 of the image processing apparatus400 generates the information about the three-dimensional image on thebasis of the model data 440 a and the control information, and causesthe image display unit 63 to display and move the three-dimensionalimage (Step S254).

When the process is continued (Step S255, Yes), the image processingapparatus 400 proceeds to Step S251. On the other hand, when the processis not continued (Step S255, No), the image processing apparatus 400finishes the process.

<<3.3. Effects of Image Processing Apparatus According to ThirdEmbodiment»

As described above, the image processing apparatus 400 receives thecontrol information indicating the movement of the terminal device 70,from the terminal device 70, and displays the three-dimensional image ofthe virtual object 11 b on the display 3 b on the basis of the modeldata 440 a and the control information. This configuration makes itpossible to reproduce the movement of the terminal device 70 held by theuser 10 positioned at a place different from a place where the user 15is positioned, with the three-dimensional image displayed by the imageprocessing apparatus 400.

4. Other Embodiments

(Shielded and Non-Shielded Areas when there is Plurality of Users)

In the example of FIG. 5 , calculation of the shielded area 60 a and thenon-shielded area 60 b that are applied to one user, by the analysisunit 130 has been described, but even if there is a plurality of users(e.g., two users), the shielded area 60 a and the non-shielded area 60 bcan be calculated. Processing units according to other embodiments willbe described using reference numerals and symbols used in FIG. 4 .

FIG. 33 is a diagram illustrating an example of a shielded area and anon-shielded area when there is a plurality of users. The analysis unit130 performs stereo matching for each portion of two images included inthe imaging information, and calculates distances from users 10A and 10Bto the display 3 and a distance from the real object 11 a to the display3, which are included in the imaging information. In the followingdescription, the distance between the user 10A and the display 3 isreferred to as a “third distance”. The distance between the user 10B andthe display 3 is referred to as a “fourth distance.” The distancebetween the real object 11 a and the display 3 is referred to as a“second distance”.

The analysis unit 130 calculates the shielded area 60 a and thenon-shielded area 60 b from a geometric relationship between the size ofthe real object 11 a, the viewpoint positions of the users 10A and 10B,the second distance, the third distance, and the fourth distance. Theshielded area 60 a indicates an area that the user 10A or the user 10Bcannot view due to the real object 11 a, of a display area of thedisplay 3. The non-shielded area 60 b indicates an area that the user10A and the user B can view, of the display area of the display 3.

The control unit 150 performs, for example, the processing described inthe control policy P4, on the basis of the results of analyses by theanalysis unit 130. In other words, the control unit 150 performs controlto cause the three-dimensional image to be displayed in the non-shieldedarea of the image display unit 60.

Process Procedure Performed by Image Processing Apparatus According toOther Embodiments

FIG. 34 is a flowchart illustrating a process procedure performed by animage processing apparatus according to other embodiments. Asillustrated in FIG. 34 , the image processing apparatus 100 startsacquisition of the imaging information from the image capturing unit 50(Step S300). The viewpoint position detection unit 110 of the imageprocessing apparatus 100 detects each viewpoint position of a pluralityof users 10A and 10B (Step S301).

The analysis unit 130 of the image processing apparatus 100 detects thereal object 11 a held by the user 10A (or the user 10B) (Step S302). Theanalysis unit 130 calculates the positional relationship between thereal object 11 a and the virtual object 11 b (Step S303). The controlunit 150 of the image processing apparatus 100 identifies the shieldedarea and the non-shielded area (Step S304).

The control unit 150 calculates the display position included in thenon-shielded area and generates the control information (Step S305). Thedisplay image generation unit 120 generates the information about thethree-dimensional image on the basis of the model data 140 a and thecontrol information, and causes the image display unit 60 to display thethree-dimensional image in the non-shielded area (Step S306).

When the process is continued (Step S307, Yes), the image processingapparatus 100 proceeds to Step S301. On the other hand, when the processis not continued (Step S307, No), the image processing apparatus 100finishes the process.

The image processing apparatus 100 performs the above process to displaythe three-dimensional image (virtual object 11 b) in the non-shieldedarea on the basis of the viewpoint positions of the plurality of users,enabling the plurality of users to simultaneously refer to both the realobject 11 a and the three-dimensional image.

<5. Hardware Configuration>

The image processing apparatuses according to the embodiments asdescribed above are implemented by a computer 500 having a configurationas illustrated in FIG. 35 . Hereinafter, an example of the imageprocessing apparatus 100 (200, 300, or 400) according to an embodimentwill be described. FIG. 35 is a hardware configuration diagramillustrating an example of the computer 500 implementing the functionsof the image processing apparatus. The computer 500 includes CPU 501that executes various arithmetic processing, an input device 502 thatreceives an input of data from a user, and a display 503 that displays athree-dimensional image. In addition, the computer 500 includes aninterface 504 for connection to various devices. The interface 504 isconnected to a camera or the like.

A hard disk device 506 includes an image processing program 506 a. TheCPU 501 reads the image processing program 506 a and deploys the imageprocessing program to RAM 505. The image processing program 506 afunctions as an image processing process 505 a.

The processing of the image processing process 505 a corresponds toprocessing of each of the viewpoint position detection unit 110, thedisplay image generation unit 120, the analysis unit 130, and thecontrol unit 150 which are illustrated in FIG. 4 .

Note that the image processing program 506 a is not necessarily storedin the hard disk device 206 from the beginning. For example, eachprogram is stored in a “portable physical medium” such as a flexibledisk (FD), CD-ROM, DVD disk, magneto-optical disk, or IC card that isinserted into the computer 500. Then, the computer 500 may read andexecute the image processing program 506 a.

<6. Conclusion>

The image processing apparatus includes the imaging unit, the analysisunit, and the control unit. The imaging unit images the user and thereal object. The analysis unit analyzes the attitude of the real objecton the basis of the imaging information captured by the imaging unit.The control unit controls display of the image related to the realobject on the basis of the attitude of the real object. Thisconfiguration makes it possible for the user 10 to refers to thethree-dimensional image of the virtual object 11 b having the sameattitude as the attitude of the real object 11 a held by the user, thus,supporting the work of the user 10.

The analysis unit further performs processing of calculating the size ofthe real object on the basis of the imaging information, and the controlunit controls the size of the image related to the real object on thebasis of the size of the real object. This configuration makes itpossible for the user 10 to refers to the three-dimensional image of thevirtual object 11 b having the same size as that of the real object 11 aheld by the user, thus, supporting the work of the user 10.

The analysis unit calculates the first distance between the displayscreen on which the image is displayed and the user and the seconddistance between the display screen and the real object, on the basis ofthe imaging information, and the control unit adjusts the size of theimage related to the real object, on the basis of the first distance andthe second distance. Even if the real object 11 a is positioned closerto the user 10 than to the image display unit 60, this configurationmakes it possible for the user 10 to refer to the three-dimensionalimage of the virtual object 11 b having an apparent size the same as thesize of the real object 11 a held by the user, thus, supporting the workof the user 10.

The real object is positioned between the display screen on which theimage is displayed and the user, the analysis unit calculates theshielded area that the user cannot refer to due to the real object andthe non-shielded area that the user can refer to, of the area of thedisplay screen, on the basis of the imaging information, and the controlunit causes the image related to the real object to be displayed in thenon-shielded area.

The real object is positioned between the display screen and a pluralityof users, the analysis unit calculates the shielded area that theplurality of users cannot refer to due to the real object and thenon-shielded area that the plurality of users can refer to, of the areaof the display screen, on the basis of the imaging information, and thecontrol unit causes the image related to the real object to be displayedin the non-shielded area. This configuration makes is possible for theplurality of users to simultaneously refer to both the real object 11 aand the three-dimensional image.

The control unit causes the image related to the real object to bedisplayed in the non-shielded area or the shielded area, on the basis ofthe distance between the real object and the image related to the realobject displayed on the display screen. This configuration makes itpossible to support the work of the user 10, both for the “viewing thereal object 11 a and the three-dimensional image in an overlappingmanner” and the “viewing the real object 11 a and the three-dimensionalimage in an aligned manner”.

The analysis unit calculates the movement of the position of the userand the movement of the real object, on the basis of the imaginginformation, and the control unit controls the position of the displayscreen on which the image related to the real object is displayed, onthe basis of the movement of the position of the user and the movementof the real object. This configuration makes it possible to show thechange corresponding to the movement of the real object 11 a by movingthe display position of the virtual object 11 b instead, supporting thework of the user 10.

The control unit receives, from another device, display controlinformation that is created on the basis of imaging information capturedby another imaging unit imaging another real object and that controlsthe display of the image related to the real object, and controls thedisplay of the image related to the real object on the basis of thereceived display control information. This configuration makes itpossible to reproduce the movement of the real object 11 a held by theuser 10 positioned at a place different from a place where the user 15is positioned, with the three-dimensional image displayed by the imageprocessing apparatus 300.

The control unit receives, from the terminal device, the display controlinformation that is created on the basis of the movement of the terminaldevice and that controls the display of the image related to the realobject, and controls the display of the image related to the real objecton the basis of the received display control information. Thisconfiguration makes it possible to reproduce the movement of theterminal device 70 held by the user 10 positioned at a place differentfrom a place where the user 15 is positioned, with the three-dimensionalimage displayed by the image processing apparatus 400.

The analysis unit analyzes the attitude and movement of the real object,on the basis of the imaging information, and stores the features of thereal object and information about the attitude and movement of the realobject in the storage unit. The control unit acquires the informationabout the attitude and movement corresponding to the features of thereal object, analyzed by the analysis unit, from the storage unit, andcontrols the display of the image related to the real object, on thebasis of the acquired information about the attitude and movement. Thisconfiguration makes it possible to reproduce the movement of the realobject 11 a in the past with the virtual object 11 b.

The analysis unit estimates the illumination position and theillumination color for the real object, on the basis of the imaginginformation, and the control unit causes the color of the image relatedto the real object to be changed on the basis of the illuminationposition and the illumination color. This configuration makes itpossible to reflect the effects of the illumination similar to those inilluminating the real object 11 a, in the virtual object 11 b.

It should be noted that the effects described herein are merelyexamples, and the present invention is not limited to these effects andmay have other effects.

Note that the present technology can also employ the followingconfigurations.

(1)

An image processing apparatus including:

an imaging unit that images a user and a real object;

an analysis unit that analyzes an attitude of the real object based onimaging information captured by the imaging unit; and

a control unit that controls display of an image related to the realobject based on the attitude of the real object.

(2)

The image processing apparatus according to (1), wherein the analysisunit further performs processing of calculating a size of the realobject based on the imaging information, and the control unit controls asize of the image related to the real object based on the size of thereal object.

(3)

The image processing apparatus according to (1) or (2), wherein theanalysis unit calculates a first distance between a display screen onwhich the image is displayed and the user and a second distance betweenthe display screen and the real object, based on the imaginginformation, and the control unit adjusts the size of the image relatedto the real object, based on the first distance and the second distance.

(4)

The image processing apparatus according to (1), (2), or (3), whereinthe real object is positioned between the display screen on which theimage is displayed and the user, the analysis unit calculates a shieldedarea that the user cannot refer to due to the real object and anon-shielded area that the user can refer to, of an area of the displayscreen, based on the imaging information, and the control unit causesthe image related to the real object to be displayed in the non-shieldedarea.

(5)

The image processing apparatus according to any one of (1) to (4),wherein the real object is positioned between the display screen and aplurality of users, the analysis unit calculates a shielded area thatthe plurality of users cannot refer to due to the real object and anon-shielded area that the plurality of users can refer to, of the areaof the display screen, based on the imaging information, and the controlunit causes the image related to the real object to be displayed in thenon-shielded area.

(6)

The image processing apparatus according to (4), wherein the controlunit causes the image related to the real object to be displayed in thenon-shielded area or the shielded area, based on a distance between thereal object and the image related to the real object displayed on thedisplay screen.

(7)

The image processing apparatus according to any one of (1) to (6),wherein the analysis unit calculates movement of a position of the userand movement of the real object, based on the imaging information, andthe control unit controls a position of the display screen on which theimage related to the real object is displayed, based on the movement ofthe position of the user and the movement of the real object.

(8)

The image processing apparatus according to (1), wherein the controlunit receives, from another device, display control information that iscreated based on imaging information captured by another imaging unitimaging another real object and that controls the display of the imagerelated to the real object, and controls the display of the imagerelated to the real object based on the received display controlinformation.

(9)

The image processing apparatus according to (1), wherein the controlunit receives, from the terminal device, display control informationthat is created based on movement of a terminal device and that controlsthe display of the image related to the real object, and controls thedisplay of the image related to the real object based on the receiveddisplay control information.

(10)

The image processing apparatus according to (1), wherein the analysisunit further performs processing of analyzing an attitude and movementof the real object, based on the imaging information, and storingfeatures of the real object and information about the attitude andmovement of the real object, in a storage unit.

(11)

The image processing apparatus according to (1), wherein the controlunit acquires information about an attitude and movement correspondingto features of the real object, analyzed by the analysis unit, from astorage unit, and controls the display of the image related to the realobject, based on the acquired information about the attitude andmovement.

(12)

The image processing apparatus according to any one of (1) to (7),wherein the analysis unit estimates an illumination position and anillumination color for the real object, based on the imaginginformation, and the control unit causes the color of the image relatedto the real object to be changed based on the illumination position andthe illumination color.

(13)

The image processing apparatus according to any one of (1) to (12),wherein the control unit controls display of a three-dimensional imagerelated to the real object.

(14)

An image processing method including performing the following processesby a computer:

analyzing an attitude of a real object based on imaging informationcaptured by an imaging unit imaging a user and the real object; andcontrolling display of an image related to the real object based on theattitude of the real object.

(15)

An Image Processing Program

causing a computer to function as:

an analysis unit that analyzes an attitude of a real object based onimaging information captured by an imaging unit imaging a user and areal object; and

a control unit that controls display of an image related to the realobject based on the attitude of the real object.

REFERENCE SIGNS LIST

-   -   50, 51, 52, 53 IMAGE CAPTURING UNIT    -   60, 61, 62, 63 IMAGE DISPLAY UNIT    -   100, 200, 300, 400 IMAGE PROCESSING APPARATUS    -   110, 210, 310, 410 VIEWPOINT POSITION DETECTION UNIT    -   120, 220, 320, 420 DISPLAY IMAGE GENERATION UNIT    -   130, 230, 330, 430 ANALYSIS UNIT    -   140, 240, 340, 440 STORAGE UNIT    -   140 a, 240 a, 340 a, 440 a MODEL DATA    -   140 b, 240 b, 340 b, 440 b ANALYSIS RESULT TABLE    -   150, 250, 350, 450 CONTROL UNIT    -   260, 360, 460 COMMUNICATION UNIT

1. An image processing apparatus including: an imaging unit that imagesa user and a real object; an analysis unit that analyzes an attitude ofthe real object based on imaging information captured by the imagingunit; and a control unit that controls display of an image related tothe real object based on the attitude of the real object.
 2. The imageprocessing apparatus according to claim 1, wherein the analysis unitfurther performs processing of calculating a size of the real objectbased on the imaging information, and the control unit controls a sizeof the image related to the real object based on the size of the realobject.
 3. The image processing apparatus according to claim 1, whereinthe analysis unit calculates a first distance between a display screenon which the image is displayed and the user and a second distancebetween the display screen and the real object, based on the imaginginformation, and the control unit adjusts the size of the image relatedto the real object, based on the first distance and the second distance.4. The image processing apparatus according to claim 1, wherein the realobject is positioned between the display screen on which the image isdisplayed and the user, the analysis unit calculates a shielded areathat the user cannot refer to due to the real object and a non-shieldedarea that the user can refer to, of an area of the display screen, basedon the imaging information, and the control unit causes the imagerelated to the real object to be displayed in the non-shielded area. 5.The image processing apparatus according to claim 4, wherein the realobject is positioned between the display screen and a plurality ofusers, the analysis unit calculates a shielded area that the pluralityof users cannot refer to due to the real object and a non-shielded areathat the plurality of users can refer to, of the area of the displayscreen, based on the imaging information, and the control unit causesthe image related to the real object to be displayed in the non-shieldedarea.
 6. The image processing apparatus according to claim 4, whereinthe control unit causes the image related to the real object to bedisplayed in the non-shielded area or the shielded area, based on adistance between the real object and the image related to the realobject displayed on the display screen.
 7. The image processingapparatus according to claim 1, wherein the analysis unit calculatesmovement of a position of the user and movement of the real object,based on the imaging information, and the control unit controls aposition of the display screen on which the image related to the realobject is displayed, based on the movement of the position of the userand the movement of the real object.
 8. The image processing apparatusaccording to claim 1, wherein the control unit receives, from anotherdevice, display control information that is created based on imaginginformation captured by another imaging unit imaging another real objectand that controls the display of the image related to the real object,and controls the display of the image related to the real object basedon the received display control information.
 9. The image processingapparatus according to claim 1, wherein the control unit receives, fromthe terminal device, display control information that is created basedon movement of a terminal device and that controls the display of theimage related to the real object, and controls the display of the imagerelated to the real object based on the received display controlinformation.
 10. The image processing apparatus according to claim 1,wherein the analysis unit further performs processing of analyzing anattitude and movement of the real object, based on the imaginginformation, and storing features of the real object and informationabout the attitude and movement of the real object, in a storage unit.11. The image processing apparatus according to claim 1, wherein thecontrol unit acquires information about an attitude and movementcorresponding to features of the real object, analyzed by the analysisunit, from a storage unit, and controls the display of the image relatedto the real object, based on the acquired information about the attitudeand movement.
 12. The image processing apparatus according to claim 1,wherein the analysis unit estimates an illumination position and anillumination color for the real object, based on the imaginginformation, and the control unit causes the color of the image relatedto the real object to be changed based on the illumination position andthe illumination color.
 13. The image processing apparatus according toclaim 1, wherein the control unit controls display of athree-dimensional image related to the real object.
 14. An imageprocessing method including performing the following processes by acomputer: analyzing an attitude of a real object based on imaginginformation captured by an imaging unit imaging a user and the realobject; and controlling display of an image related to the real objectbased on the attitude of the real object.
 15. An image processingprogram causing a computer to function as: an analysis unit thatanalyzes an attitude of a real object based on imaging informationcaptured by an imaging unit imaging a user and a real object; and acontrol unit that controls display of an image related to the realobject based on the attitude of the real object.